Tencent Text-Video Retrieval: Hierarchical Cross-Modal Interactions with Multi-Level Representations

Text-Video Retrieval plays an important role in multi-modal understanding and has attracted increasing attention recent years. Most existing methods focus on constructing contrastive pairs between whole videos complete caption sentences, while overlooking fine-grained cross-modal relationships, e.g., clip-phrase or frame-word. In this paper, we propose a novel method, named Hierarchical Cross-Modal Interaction (HCMI), to explore multi-level relationships among video-sentence, clip-phrase, frame-word for text-video retrieval. Considering intrinsic semantic frame relations, HCMI performs self-attention frame-level correlations adaptively cluster correlated frames into clip-level video-level representations. way, constructs video representations frame-clip-video granularities capture content, text at word-phrase-sentence the modality. With text, hierarchical learning is designed i.e., frame-word, which enables achieve comprehensive comparison modalities. Further boosted by adaptive label denoising marginal sample enhancement, achieves new state-of-the-art results various benchmarks, Rank@1 of 55.0%, 58.2%, 29.7%, 52.1%, 57.3% MSR-VTT, MSVD, LSMDC, DiDemo, ActivityNet, respectively.